The utility of futility.

See related article, pages 2410–2414. In this issue, Palesch et al 1 discuss the single-arm, phase II futility study design and illustrate how its use might have avoided 3 large (and costly) but negative phase III therapeutic trials for ischemic stroke patients. The authors offer strong arguments to support their conclusion that use of this design as a strategy in phase II development “could permit the testing of a wider array of promising treatments at a fraction of the cost of taking all treatments directly to phase III trials.” In a nutshell, they argue that there is utility in futility testing. Although common in early-phase oncology trials, the futility study (singleor double-armed) may be less familiar to readers of this journal, and careful scrutiny of the design, especially of the formulation of null and alternative hypotheses, is worthwhile. Briefly, in a futility study, the null hypothesis states that the experimental therapy is sufficiently promising to warrant definitive, phase III testing, whereas the alternative hypothesis states that the experimental therapy lacks the prespecified superiority. Thus, the futility design reverses the logical status of null and alternative hypotheses as most often formulated in the traditional efficacy design. Whereas in the latter design, sufficient evidence is required to declare a therapeutic effect statistically significant, in the futility design, there is a presumption of benefit, and sufficient evidence is required to declare a significant shortfall from that benefit, such that it would be futile to proceed to large-scale testing with the given therapy. The authors argue that this formulation, with its null presumption of benefit, is appropriate in phase II research on the grounds that of the 2 types of error that can be committed—declaring a truly superior therapy futile or declaring a truly nonsuperior therapy worthy of continued testing—the former is the more important. One should therefore view it as a type I error with appropriate control of the error rate ( ). This the futility design accomplishes. I suspect the main attraction of the authors’ proposal will be the relatively small number of patients required to conduct the experiment in comparison to the sample sizes required for the traditional phase III randomized, controlled trial. Indeed, the authors illustrate 5-fold, 10-fold, and even larger potential reductions in sample size. Three key elements serve to achieve this efficiency. The first is the one-sided nature of the hypotheses (the null states there is a real therapeutic benefit; the alternative denies this directional superiority). The second key element is the use of somewhat more liberal values of alpha, eg, 0.10 (1-tailed), compared with the conventional 0.05 level (2-tailed, or 0.025 1-tailed). This is arguably appropriate for phase II testing. The third key element, which is the most influential and perhaps the most controversial, is the use of only a single (experimental) arm. For this, one needs a clear clinical notion of benefit, based on historical– control data, to quantify the “minimally worthwhile improvement” required for the single-arm futility design. The authors focus on the single-arm design, but there is no intrinsic reason why the futility design cannot be applied with a concurrent control arm. Indeed, the ongoing NINDS-funded QALS study of high-dose coenzyme Q10 in patients with amyotrophic lateral sclerosis2 uses a 2-arm futility design. Although this is not the appropriate forum to debate the pros and cons of single-arm studies, the following points may help to avoid some pitfalls of the single-arm futility design: